Method for image processing of image data for image and visual effects on a two-dimensional display wall

ABSTRACT

A captured scene captured of a live action scene while a display wall is positioned to be part of the live action scene may be processed. To perform the processing, image data of the live action scene having a live actor and the display wall displaying a first rendering of a precursor image is received. Further, precursor metadata for the precursor image displayed on the display wall and display wall metadata for the display wall is determined. An image matte is accessed, where the image matte indicates a first portion associated with the live actor and a second portion associated with the precursor image on the display wall in the live action scene. Pixel display values to add or modify an image effect or a visual effect are determined, and the image data is adjusted using the pixel display values and the image matte.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to and the benefit of U.S. ProvisionalPatent Application No. 63/288,108, filed on Dec. 10, 2021, the contentsof which are hereby incorporated by reference in their entirety.

FIELD

The present disclosure generally relates to computer image processingfor image data, including stereoscopic image data captured by two ormore cameras, and more particularly to processing image data having atwo-dimensional (2D) display wall with one or more three-dimensional(3D) objects placed relative to the 2D display wall.

BACKGROUND

In computer-generated image generation and animation, of images and/orvideo sequences, there might be a desire to incorporate a digital wallwhile capturing a live action scene of an actor. In a detailed liveaction scene that incorporates background animated elements, it could bedifficult to properly coordinate a live actor with those backgroundelements. Furthermore, if it is desirable to project lights, colors, orother real-world effects on a live actor, it may be tedious to ensurethat those effects are properly aligned between the live actor and theanimated background imagery that is later added to a scene. Animatedobjects that may be placed in a background scene and/or with a liveaction scene can comprise many individual objects, which may have theirown lighting effects, colors, and/or interactions with live actors. Forexample, a scene involving an explosion or other intense light may havefeatures that cause colors to be projected onto live actors. Backgroundscenes may also involve stage elements and/or creatures that interactwith live actors, such as by acting as an environment and/or engagingwith live actors.

For simple scenes and/or backgrounds, modeling or drawing individualbackground objects and/or scenes might not be difficult. However, asviewers have come to expect more complex visuals, there is a need for aprocedural processing, rendering, and adjusting backgrounds to appearmore realistic. Further, stereoscopic imaging may be used to capturescenes as they would be viewed from different angles, and therefore adddepth and 3D elements to the captured images and video. When 2D elementsare added to a display wall or other 2D object, the display wall may notappear realistic with 3D live actors when a live scene isstereoscopically captured. Thus, it may be desirable to adjust and/orre-render background imagery captured of a 2D wall in real-time and/orafter capturing a live action scene in order to provide more realistic3D visuals.

SUMMARY

According to some embodiments, a computer-implemented method forprocessing, in an image processing system, a captured scene captured ofa live action scene while a display wall is positioned to be part of thelive action scene, wherein the display wall comprises one or morestructures capable of displaying imagery, the method may includereceiving image data of the live action scene having a live actor andthe display wall displaying a first rendering of a precursor image,wherein the image data is captured by at least one camera in at leastone placement relative to the live actor and the display wall,determining precursor metadata for the precursor image displayed on thedisplay wall and display wall metadata for the display wall, wherein theprecursor metadata comprises pixel display data for display wall pixelsof the display wall, and wherein the display wall metadata comprisesgeometry data for a display wall position relative to the at least onecamera, accessing an image matte for the image data, wherein the imagematte indicates a first portion associated with the live actor and asecond portion associated with the precursor image on the display wallin the live action scene, determining pixel display values to add ormodify at least one of an image effect from the display wall pixels ofthe precursor image or a visual effect provided in the live actionscene, and wherein the pixel display values comprise one or moreadjustments to pixels in the image data, and adjusting the image data ofthe captured scene using the pixel display values and the image matte toadd or modify the at least one of the image effect or the visual effectindependent of rendering of the precursor image on the display wallduring capture of the image data.

This Summary is provided to introduce a selection of concepts in asimplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tolimit the scope of the claimed subject matter. A more extensivepresentation of features, details, utilities, and advantages of thesurface computation method, as defined in the claims, is provided in thefollowing written description of various embodiments of the disclosureand illustrated in the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIG. 1 illustrates an exemplary 2D display wall that includes pixelsdisplaying a precursor image for use with a live action scene, in anembodiment.

FIG. 2 illustrates an exemplary live action scene having stereoscopiccameras capturing a live actor in a 3D environment with a 2D displaywall, in an embodiment.

FIG. 3 illustrates an exemplary environment for a live action scenecaptured by a left oriented camera, in an embodiment.

FIG. 4 illustrates an exemplary environment for a live action scenecaptured by a right oriented camera, in an embodiment.

FIG. 5 is a flowchart of an exemplary method as might be performed by acomputing system when generating an image matte for stereoscopic imagedata captured by stereoscopic cameras of a 3D object and a 2D displaywall, in an embodiment.

FIG. 6 is a flowchart of an exemplary method as might be performed by acomputing system when rendering and/or adjusting pixels for use withimage data of live action scenes having a 3D object and a 2D displaywall, in an embodiment.

FIG. 7 illustrates a system for processing precursor images displayed ona 2D display wall during capture of a live action scene, in anembodiment.

FIG. 8 illustrates an example visual content generation system as mightbe used to generate imagery in the form of still images and/or videosequences of images, according to various embodiments.

FIG. 9 is a block diagram illustrating an example computer system uponwhich computer systems of the systems illustrated in FIGS. 1 and 8 maybe implemented.

DETAILED DESCRIPTION

In the following description, various embodiments will be described. Forpurposes of explanation, specific configurations and details are setforth in order to provide a thorough understanding of the embodiments.However, it will also be apparent to one skilled in the art that theembodiments may be practiced without the specific details. Furthermore,well-known features may be omitted or simplified in order not to obscurethe embodiment being described.

Computer simulation that is used with live actors may be placed into alive action scene in different ways. Conventionally, live actors may actin front of a green screen or other colored background that allows forchroma keying to provide visual effects in post-production and after alive action scene is captured. This may be used to provide differentbackground effect, which allows for providing computer simulatedeffects, backgrounds, and the like with live actors. However, chroma keycompositing may suffer from issues of realism when adding visual effectsin post-production. For example, the green screen does not include anycomputer simulation and/or animation when the live actor is acting inthe live action scene. Thus, the live actor may be required to pretendthat certain animated portions of the scene are present. This may be anissue where those animated elements are interacted with by the liveactor, such as an environmental object and/or character. A live actormay not know exactly where a cliff or edge may be when later added asanimated elements or may not know the exact location of a character orcreature that the live actor is engaging with in the live action scene.Further, the green screen used for chroma keying does not providelighting or colors, which may be projected onto the live actor (e.g., inthe case of an explosion or the like) or may be reflected in the liveactors' eyes, glasses, wardrobe pieces, or the like.

In this regard, a display wall may be used, which may include a displayscreen (e.g., an LED, LCD, LED LCD, OLED, or the like) that is capableof outputting a rendering of an image or a video. The rendering maycorrespond to a precursor image, which may be an image from a rendereror a compositor. This image may be entirely or partially computergenerated and/or animated, may be captured earlier from a live actionscene, or a combination thereof. The precursor image may be a singleimage displayed on the display wall or may be a sequence of images, suchas frames of a video or animation. The precursor image may includeprecursor metadata for computer generated imagery and/or pixel displaydata for pixels of the display wall. In this regard, the precursormetadata may include output pixels, data, color, intensity, and the likefor outputting the image on the display wall.

The display wall may correspond to one or more structures that are thenpositioned in the real-world live action scene, and may be planar,curved, or the like. In this regard, the display wall may serve as abackground, however, this may not be the only orientation and thedisplay wall may also be placed above, next to, or otherwise orientedwith regard to a live actor. The display wall may be placed relative toa scene or stage element, such as a live actor or other object in the 3Dreal-world scene for the live action. The live actor then acts andinteracts with the live action scene corresponding to the real-worldenvironment that is being captured by one or more cameras. The liveaction scene may also be captured by other sensors and/or sensingdevices, including optical sensing devices, depth sensors and/or rangingsensors (e.g., LiDAR), and the like.

When the live actor interacts with and/or performs in the live actionscene, the display wall may output one or more precursor images that maybe used with the live actor to capture image data. This image data maybe captured by one or more or more cameras. For example, a camera may beoriented relative to the 2D display wall to capture background imageryand pixels of the 2D display wall while stage elements are present in alive action scene in front of and/or relative to the 2D display wall. Insome embodiments, the image data may be stereoscopically captured by twoor more cameras may be placed proximate to each other in differentlocations so that the cameras may capture different angles of the liveaction scene. In some embodiments, the cameras may be oriented so as tomimic how a person or creature may view the live action scene, such ashow a human that would view the live action scene. This may be used toprovide additional realism to capturing the live action scene and/orcapture the live action scene in 3D so that the image data may later berendered and output in 3D when viewed by an audience.

However, when using a 2D display wall with a 3D live actor and/or 3Dobjects in front of the display wall, different capturing of the liveaction scene, live actor, 3D objects, and/or 2D display wall may causedifferent cameras to capture different versions of the live actionscene. This may be caused when different cameras may capture differentangles of the live action scene so that different parts and/orappearances of the display wall are captured behind the live actor orother 3D objects. The actor and/or objects in front of the display wallmay block or obstruct certain pixels of the display wall based ondifferent angles, while capturing different pixels from differentangles. Thus, the live action scene may be captured using one or morecameras and/or other sensors while the display wall is present in thescene and outputting one or more precursor images or other precursorimagery.

The live actor may be engaging with the display wall and/or otherelements and objects in the live action scene while computer generatedimagery is being generated by a renderer and displayed on the displaywall. The display wall may also emit or output light, colors, and thelike that are projected on and/or reflected by the live actor or otherobjects in the live action scene. An image processor may, in real-time,near real-time, or at a later post-processing time, determine whichportions of the captured image data correspond to the live actor, andwhich portions correspond to the display wall (e.g., background ordisplay wall pixels). The portions may correspond to individual pixelsof the display wall and/or live actor in the image data and may bedifferent between different image data captured by each camera. Theimage processor may then generate an image mat, which may be generatedand/or stored as a pixel array where values or data for pixels in thepixel array indicate whether the corresponding pixel is part of thedisplay wall or another foreground actor, object, or the like in thelive action scene.

The image matte may then be used to selectively modify pixels in theprecursor metadata for the displayed precursor image to change theoutput or rendering of those pixels when displayed on the display walland/or captured in the stereoscopic image data. In some embodiments, theimage matte may be a binary mat, a junk matte (e.g., where pixels areidentified as part of the display wall, not part of the display wall,and/or uncertain as to whether those pixels belong to the display wall).For example, the image matte may be used to modify pixels thatcorrespond to the display wall when a precursor image is being displayedon the display wall or at a later time when the stereoscopic image datais processed and background pixels are moved, warped, adjusted, and/orre-rendered. Thus, the image matte may correspond to an alpha channelimage that may allow for modification, adjustment, and/or change of oneor more portions of the stereoscopic image data (e.g., the live actor,the display wall, or another portion of the image data).

In this regard, the precursor metadata may include a scene descriptionwith information about the 3D position of a computer-generated objects,characters, or the like in the precursor image for the display wall.Modification of pixels corresponding to the display wall may includereplacement of pixel color values and/or pixel outputs that correspondto the display wall to the pixel values that would have actually beencaptured if the precursor image on the display wall actually existed inthe live action scene and real-world 3D environment. This may allowstereoscopic cameras to capture stereo image pairs of the live actor infront of the display wall but modified so that the captured image dataappears as if the precursor image was actually present in 3D instead of2D on the display wall. Thus, a renderer and/or compositor system mayperform pixel replacement corresponding to the display wall, taking intoaccount precursor metadata of the precursor image on the display wall,to present a processed precursor image on the display wall for a 3Dimage in a live action scene.

The image processor may generate the image matte using differenttechniques. For example, the image processor may determine the imagematte by computing the image matte, from the precursor image metadataand/or display wall metadata (e.g., information for placement of thedisplay wall, depths to 3D stage elements and/or cameras, lighting oreffects in the real-world environment, etc.) with the image data basedon what the camera should have captured, and what the camera actuallycaptured. Pixel values that are the same or similar to what should havebeen captured in the region those pixels should be located may beidentified as background pixels, while other pixels may correspond to alive actor or other 3D object in the live action scene. Further, theimage processor may determine where the display wall is in a scene, suchas a distance or depth from the cameras to the display wall, to identifythose pixels that correspond to the display wall and those thatcorrespond to a foreground object or live actor.

Once the image processor determines the image mat, the image processormay execute different processes with the image mat. For example, theimage processor may move the background pixels to stereo-displacedlocations and/or so that the pixels are now stereo-displaced pixels ofthe precursor image that appear as though the precursor image is in 3Dwhen captured by stereoscopic cameras. This may include adjusting,moving, and/or warping pixels of the precursor image from a cameraposition and/or camera settings with the precursor metadata and/ordisplay wall metadata. This may be based on the position of the actualcamera and display wall at the time of capture and the adjustments orreplacement pixels may be generated using the image matte to appear as a3D image or imagery on the display wall. Depth may also be determinedusing one or more depth sensors for pixel identification andreplacement. Further, this may account for different types of camerascapturing the scene, such as a “hero” camera for a main camera capturingthe important or main elements of the live action scene (e.g., the liveactor), as well as non-hero cameras.

Although the pixels are referred to as replacement pixels, the pixelsmay instead be data and/or metadata that cause changing, warping,adjusting, or moving original background pixels associated with thedisplay wall instead of directly replacing such pixels. For example,adjusting pixels may include moving pixels within image data captured ofthe display wall in the live action scene and/or on the display walldirectly. Movements may correspond to moving the pixels in one or more2D or 3D directions for the display wall and/or live action scene, aswell as computing other changes to positions and/or locations for pixelsin image data. Adjusting pixels may also include warping or changingpixels, which may change pixel values or data (e.g., color, brightness,luminosity, effect, intensity, and/or the like) instead of or inaddition to direct movement of the pixels. Thus, pixel adjustments mayinclude changes to pixel values and data in image data instead of or inaddition to direct replacement of pixels with another or differentpixel.

For example, with stereoscopic image data captured by two or morecameras, pixels from each camera may not exactly match in the resultingimage data. A left oriented camera may capture a different scene than aright orientated camera. These differences in the pixels captured of adisplay wall may be slight, such as where the two cameras are nearby inorder to attempt to mimic eye placement of a character or creatureviewing the live action scene having the live actor or other stageelement and the display wall that is 2D in the live action scene.However, these differences in the pixels of the display wall may also belarger with different camera placements. In order to compensate forthese differences and view a stereoscopic image on the 2D display wall,the pixels corresponding to the 2D display wall in one or more cameraimages and data from the two or more stereoscopic cameras may be movedand/or warped in the resulting image data. This may include movingbackground pixels for the display wall to compensate for one or more ofthe other cameras and/or warping those pixels to different placements inthe resulting image data (e.g., the stereoscopic image data frommultiple cameras). These placements for moving or warping pixels may bean average between each of the cameras or may be compensated to a largerdegree to or away from one or more of the cameras. In some embodiments,the placements may be moved or warped to correspond directly to theimage data from another camera, but other lesser placements may bedetermined between distance differences of the captured backgroundpixels between the two or more cameras.

In further embodiments, the background pixels may be replaced togenerate a higher resolution image on the display wall, such as if thecameras capture a lower resolution image based on distances to thedisplay wall, visual effects, camera lenses or components, camera focus,and/or the like. The background pixels may be replaced in a fast mannerin real-time or near real-time so that the precursor image may becaptured with the live actor in the live action scene and may be used bythe live actor when interacting and performing in the live action scene.This may be done in a first and/or lower quality so that the precursorimage appears as a 3D image, and may later further be performed in asecond, slower, and/or higher quality during post-processing of theprecursor image on the display wall. The image matte may also be used toapply an image, real-world, and/or visual effect that is applied to thestereoscopic image data. This may be applied not to the captured lightfrom the display wall but instead to the precursor metadata andsubstituted for the captured live action image of the display wall. Thismay include adjusting depths of the corresponding effect in order toprovide changes to depths of the effect, distances of the effect,thickness or density of the effect, and the like. For example, differenteffects may include an environmental effect, a lighting effect, anelemental effect, a venue-based effect, a blurring effect, a focuseffect, an increased resolution effect, or a color correction. Effectssuch as rain, snow, fog, or the like may be dependent on depth, as wellas other visual effects from lighting and the like. In this manner, amore realistic and 3D imagery may be rendered on the display wall toautomatically provide enhanced visual effects and better capture imagedata, while reducing user input, post-processing adjustments, andadditional data processing.

FIG. 1 illustrates an exemplary 2D display wall that includes pixelsdisplaying a precursor image for use with a live action scene, in anembodiment. Environment 100 of FIG. 1 includes a display wall 102 thatmay be located in a real-world environment where one or more live actorsmay engage with display wall 102 while acting, performing, and/or beingcaptured by one or more cameras. In this regard, display wall 102 inenvironment 100 may display one or more precursor images while berecorded and captured by the camera(s) in a live action scene.

For example, display wall 102 may correspond to a liquid crystal display(LCD), a light-emitting diode (LED) display, plasma display, acombination thereof, or the like (including LCD LED, thin-filmtransistor (TFT) LCD, OLED, etc.). In this regard, display wall 102 mayconsist of and/or include pixels 104, which may be used to emit light ofcertain colors, intensities, and other parameters when outputting anddisplaying one or more images. Pixels 104 may be picture or imageelements that may correspond to the smallest point or controllableelement of display wall 102 that may make up imagery displayed ondisplay wall 102. This may be used to display scene 106 from a renderer,which may correspond to computer-generated imagery, early captured orrecorded live action scenes and/or objects, or a combination thereof. Inorder to display scene 106 on display wall 102 using pixels 104, arenderer and/or compositor may be used for image processing and output.Scene 106 may correspond to a precursor image that is displayed ondisplay wall 102. A precursor image may be one or multiple individualimages, which may be displayed in sequence, such as frames of ananimation or video. The precursor image for scene 106 may therefore beprovided by a renderer and/or may be processed using a compositor. Theprecursor image may also be associated with precursor metadata that mayinclude computer-generated imagery metadata, such as the scenedescription for scene 106 that is used in rendering and outputting scene106 on display wall 102 using pixels 104.

A renderer may correspond to hardware and/or software that generates oneor more images, or data usable for representing those images, based on ascene description, which may be single images or frame of an animationor video. A compositor may correspond to hardware and/or software thatmay combine image data of a captured live action scene and a renderedscene to form composited imagery. The captured live action scene may becaptured as image data, such as that of a live actor performed duringrecording, and live action metadata of the capturing of the scene, suchas camera settings, camera positions, lighting conditions, visualeffects, etc.

Display wall 102 may correspond to one or more structures that may bepositioned in a live action scene and be capable of displaying imageryfrom a renderer, compositor, or the like. In some instances, displaywall 102 may be a single, planar display and therefore provide a 2Doutput of scene 106. In other instances, display wall 102 may correspondto one of a plurality of planar or curved elements and/or display panelsand/or may include multiple display panels in varying positions ororientations to generate a single wall for precursor imagery. Displaywall 102 may correspond to an LED wall or other structure capable ofdisplaying imagery. Display wall 102 may correspond to a background of alive action scene, but it is not necessary display wall 102 is abackground and instead may be orientated and/or otherwise placed in alive action scene. Display wall metadata for display wall 102 maycorrespond to data that represents details of display wall 102, such asits construction, its orientation, resolution, size, etc., and itsposition in the live action scene. In some embodiments, this may bedetermined by one or more depth, distance, and/or ranging sensors, whichmay be used in combination with optical cameras and/or sensors fordistance finding between real-world 3D objects and/or display wall 102and one or more cameras.

FIG. 2 illustrates an exemplary live action scene having stereoscopiccameras capturing a live actor in a 3D environment with a 2D displaywall, in an embodiment. A display wall 202 is shown in environment 200,where environment 200 may correspond to a live action scene with anactor 208 and stereoscopic cameras 210 that capture imagery within thelive action scene. In this regard, display wall 202 may be used todisplay a precursor image having a scene 204 while actor 208 is presentand/or performing in environment 200. Scene 204 may be used to provideimagery that allows actor 208 to perform and/or interact with theimagery, such as a character or creature that actor 208 may conversewith and/or one or more stage elements that allows actor 208 todetermine their placements or performance. Further, scene 204 may beoutput by pixels 206 that may have certain light, color, and/orintensity that allows for capture, projection in the live action scene,and/or reflection on actor 208.

When capturing the live action scene in environment 200 that includesdisplay wall 202 and actor 208, stereoscopic cameras 210 may be used tocapture stereoscopic image data of the live action scene. For example, aleft camera 212 a and a right camera 212 b may correspond to two camerasof the set for stereoscopic cameras 210. Each of stereoscopic cameras210 may be oriented and/or placed relative to display wall 202 and liveactor 208 so that each of stereoscopic cameras 210 capture a differentangle of the live action scene. Left camera 212 a and right camera 212 bmay be oriented so that capture of the live action scene may appear morerealistic and/or in 3D. For example, left camera 212 a and right camera212 b may be placed to mimic vision or eyes of a character or creatureor so that they otherwise would be similar to a viewer when viewing thelive action scene. Thus, left camera 212 a is oriented at a left cameraangle 214 a while right camera 212 b is oriented at a right camera angle214 b.

However, and as noted above, different ones of pixels 206 may becaptured around actor 208 by stereoscopic cameras 210. For example, aportion of pixels 206 around live actor 208 may be hidden or may bevisible by each of left camera 212 a and right camera 212 b based onleft camera angle 214 a and right camera angle 214 b, respectively.Further, differences in placements, capture angles, and/or orientations,as well as camera construction and/or use, may cause different imagecapture of pixel values on display wall 202. When left camera 212 a isoriented at left camera angle 214 a, pixels on a left side of actor 208may be visible, such as those that may be behind live actor 208 whenviewed directly straight forward. However, left camera angle 214 aallows for viewing of certain ones of pixels 206 behind actor 208. Incontrast, portions of pixels 206 on a right side of actor 208 may behidden based on left camera angle 214 a, which may normally be visiblewhen viewing actor 208 and display wall 202 in a directly straightforward manner and/or angle.

In contrast, right camera 212 b may capture different ones of pixels 206when capturing image data of the live action scene having display wall202 with actor 208 at right camera angle 214 b. Thus, other ones ofpixels 206 may be visible or hidden when viewed and/or captured at rightcamera angle 214 b. Thus, when combining image data from each camera,the image data may be slightly different between each of stereoscopiccameras 210. In order to combine the image data and properly renderand/or output stereoscopic image data in a more realistic and 3D manner,pixels constituting display wall may be adjusted, as discussed herein,to provide movements, warping, replacement pixels, and/or pixel valuesfor adjusting one or more precursor images on display wall 202. Thus,stereoscopic image data captured by stereoscopic cameras may be adjustedusing an image matte that distinguishes display wall 202 from actor 208.Adjustment of pixel data and values for scene 204 on display wall 202may be performed as discussed in further detail with regard to FIGS. 3-6.

FIGS. 3 and 4 demonstrate different camera angles of a stereoscopiccamera set, which may capture different background pixels of a displaywall when capturing stereoscopic image data of a live action scenehaving the display wall with a live actor. In this regard, FIG. 3illustrates an exemplary environment for a live action scene captured bya left oriented camera, in an embodiment. FIG. 4 illustrates anexemplary environment for a live action scene captured by a rightoriented camera, in an embodiment. FIGS. 3 and 4 are discussed belowwhen performing operations to generate and use an image matteidentifying the live actor and the display wall for adjusting backgroundpixels of the display wall for more realistic imagery on the displaywall.

Environment 300 of FIG. 3 includes a display wall 302 and environment400 of FIG. 4 includes a display wall 402. Display walls 302 and 402 maycorrespond to the same display wall, such as display wall 202 which isbeing captured by stereoscopic cameras 210. However, in otherembodiments, display walls 302 and 402 may correspond to differentdisplay walls and/or a display wall displaying different precursorimages, such as when cameras are moved and/or a camera is rotated toanother position and/or angle with respect to a display wall. In thisregard, an actor 304 in environment 300 and an actor 404 in environment400 may be placed relative to display wall 302 and display wall 402,respectively, such as by being positioned in front of display walls 302and 402 when performing in a live action scene. Display walls 302 and402 may correspond to a generally planar or curved 2D surface thatdisplays precursor imagery that may be output by a renderer for a liveaction scene. Actors 304 and 404 may correspond to 3D characters, butmore generally other 3D creatures, objects, and the like may also beplaced relative to display walls 302 and 402 in a live action scene thatis captured by stereoscopic cameras.

In environment 300, a camera angle 306 captures image data of displaywall 302 and actor 304 from a left angle, for example, as may becaptured by left camera 212 a from left camera angle 214 a. Camera angle306 may be angled and/or positioned relative to display wall 302 andactor 304 so that different pixels on display wall 302 are capturedrelative to the live scene captured in environment 400 of FIG. 4 . Forexample, camera angle 306 captures left-side pixels 308 a and 308 b andright-side pixels 310 a and 310 b. If actor 304 and display wall areviewed in a straight line (e.g., where the camera or viewer is alignedalong a straight line with a center of actor 304 to a center of displaywall 302), certain pixels of display wall may be viewed and captured,while actor 304 may hide, obscure, or block other pixels that aredirectly behind actor 304.

However, those blocked pixels may be changed, which may be slight or maybe significant, based on camera angle 306 when camera angle 306 isangled relative to actor 304 in front of display wall 302. For example,camera angle may be positioned and angled by a small amount relative toa straightforward direction (e.g. 5 or 10 degrees), or may besignificantly angled (e.g., by 45 degrees). Left-side pixels 308 a and308 b in environment 300 may therefore include pixels on display wall302 that are not normally viewable behind actor 304 when viewed from astraightforward direction. However, right-side pixels 310 a and 310 bmay be more hidden behind actor 304, and therefore additional backgroundpixels on display wall 302 that are normally viewable behind actor 304when viewed from a straightforward direction may now be blocked by actor304 and hidden from view. Further, if camera angle is angled relativelyupward or downward when capturing display wall 302 and actor 304,different pixels of display wall 302 may be captured that are behindactor 304 based on the corresponding angle.

In a similar manner, camera angle 406 in environment 400 captures imagedata of display wall 402 and actor 404 at an angle and/or position thatcauses different background pixels on display wall 402 to be captured.In contrast to environment 300, camera angle 406 is angled at aright-side angle relative to display wall 402 and actor 404, which maycorrespond to right camera 212 b capturing image data from right cameraangle 214 b. In this regard, left-side pixels 410 a and 410 b may bemore hidden behind actor 404 and therefore additional background pixelson display wall 402 that are normally viewable behind actor 404 whenviewed from a straightforward direction may now be blocked by actor 404and hidden from view. However, right-side pixels 408 a and 408 b mayinstead include pixels on display wall 402 that are not normallyviewable behind actor 404 when viewed from a straightforward direction,but are now visible due to camera angle 406. Therefore, image datacaptured from each of camera angles 306 and 406 may each have differentbackground pixels on display walls 302 and 402, respectively, such asthose that may be around actors 304 and 404, respectively, based oncorresponding camera angles.

Camera angles 306 and 406 may correspond to stereoscopic cameras thatmay capture stereoscopic image data that requires processing in order toproperly coordinate and display precursor image(s) on display walls 302and 402, such as when capturing a 2D precursor image that is displayedon a 2D wall while a 3D actor is further in a live action scene relativeto the 2D wall. In this regard, in an exemplary operation to adjustand/or re-render background pixels on display walls 302 and/or 402 maybe provided, either in real-time or near real-time as a precursor imagefor the background pixels are displayed or later during apost-processing operation with image data captured of display walls 302and/or 402. For example, camera angles 306 and 406 capture differentiterations of a precursor image on display walls 302 and 402,respectively, based on their corresponding angles. This may correspondto additionally captured pixels; however, this may also correspond todifferent pixel color values and other parameters caused by cameraangles 306 and 406. In this regard, a precursor image may be attemptingto display a 3D environment, character, and/or objects, but whendisplayed by display walls 302 and 402 and captured from differentangles, may appear differently (e.g., due to different emitted light,captured pixels, and the like). Thus, the precursor image on displaywalls 302 and 402 may appear differently and pixel replacement may berequired in order to synchronize and/or cause image data (includingstereoscopic image data that may be adjusted for one or more othercamera angles) to appear more realistic with a flat 2D precursor imagedisplaying 3D stage elements and features for a live action scene.

For example, a live action scene in environments 300 and 400 may becaptured by one or more cameras from camera angles 306 and 406,respectively, as well as using additional sensors (e.g., ranging and/ordepth sensors). Display walls 302 and 402 may be present in the liveaction scene(s) and may be displaying wall imagery corresponding to oneor more precursor images. Display walls 302 and 402 may fill thebackground or constitute a portion of the background and may be used topresent a background scene, project light, and/or display imagery thatmay be projected on and/or reflected by 3D objects and/or actors (e.g.,actors 304 and 404) in the live action scene(s). In some embodiments,the light emitted by the display wall may eliminate portions of the liveaction scene including actors 304 and/or 404).

An image processor may include operations, in real-time or at a latertime, to capture and/or process the image data of the live actionscene(s). The image data may correspond to stereoscopic image data, suchas when environments 300 and 400 correspond to the same live actionscene and camera angles 306 and 406 correspond to different angles oftwo or more stereoscopic cameras capturing the live action scene at thesame time. However, this is not necessary and other image data may becaptured and processed as discussed herein. The image processor may thendetermine an image matte that identifies the portions of the capturedlive action scene that correspond to the stage element (e.g., actor304/404) and/or display wall 302/402. In some embodiments, the imagematte may correspond to a binary image matte where pixels are identifiedas one of two values and/or identifiers, one for those belonging todisplay wall 302/402 and one for those belonging to actor 304/404. Theimage matte may also correspond to a junk matte having pixels assignedas display wall 302/402, actor 304/404, or undecided whether the pixelsbelong to display wall 302/402 or actor 304/404. An n-ary matte may alsobe used to assign pixel values that may be shared by display wall302/402 and actor 304/404.

This allows the image matte to assign those pixels a particular pixel orcolor value and may be used to adjust, warp, or otherwise change thosepixels belonging to display walls 302 and/or 402, as well as thosepixels belonging to actor 304 and/or 404. The image matte may bedetermined by computing which pixels belong to display walls 302 and/or402 and which belong to actor 304 and/or 404. Computations for pixelidentification may be performed by comparing what was captured in theimage data to the pixel values for the precursor metadata, as well asprocessing using display wall metadata for the display wall in thebackground (e.g., dimensions and/or properties of the display wall,distances and/or depths to the display wall and/or between stageelements and cameras, real-world data of the live action scene havingthe display wall, and the like). Those pixels matching with theprecursor metadata and/or background wall metadata within a tolerance orrange and/or being located in an expected location or vicinity in theimage data may be considered background pixels of display walls 302and/or 402. The image matte may also or instead be determined byutilizing ranging and/or depth from the cameras to display walls 302and/or 402, as well as any other position or angle information forcameras and display walls 302 and/or 402, to determine those pixels of ashallower depth captured in the image data and those of an expecteddepth of display walls 302 and/or 402. Thereafter, foreground objects,such as actors 304 and/or 404 may be identified by the shallower pixelsand display walls 302 and/or 402 may be identified by the pixels of theexpected and/or deeper depth.

With stereoscopic image data, the image matte be used to identify thosepixels that may be viewed from different cameras, which may be used tocorrect and/or replace background pixels for image data that would be acomposite of multiple camera views. The composite imagery and/or viewmay be used to adjust the precursor image to be “stereo-displaced” inthat that precursor image may appear as though it is part of thereal-world live action scene. The image matte may correspond to a pixelarray where each value in the pixel array indicates whether acorresponding pixel is part of the image data captured of one or moreprecursor images or other imagery presented on display wall 302/402 orwhether the corresponding pixel is part of a foreground object, element,or person (e.g., actor 304/404). The image matte may then be used toprocess the image data of the live action scene and selectively modifypixels that correspond to display wall 302/402. When modifying thepixels, initial precursor metadata (e.g., metadata designated the pixelvalues and/or precursor image) of the precursor image may be used, aswell as display wall metadata, live scene metadata, and/or depth orranging data detected by one or more sensors for a depth from a camerato display wall 302/402.

For example, precursor metadata for a precursor image may include scenedescription information for a scene displayed on display wall 302/402.This may include pixel outputs or other pixel display data, however,more generally the precursor metadata may have data describing the 3Dpositioning and/or placement of objects, characters, or the like in theprecursor metadata. Thus, the 3D position of an object may change whenviewed on display wall 302/402 from different camera angles, and pixeladjustment, modification, and/or replacement may be required for imagedata captured from different angles, including stereoscopic image datacaptured from different cameras. The modification of the pixels thatcorrespond to display wall 302/402 may include replacing pixel location,color, brightness, luminosity, or other data values of those pixels thatwere captured in image data with pixel color values that would have beencaptured if the stage elements and features from the precursor metadataactually existed in their corresponding positions in the live actionscene. In some embodiments, this may allow stereoscopic cameras tocapture stereoscopic image data of a pair of images (or frames ofmultiple images for a video) of actor 304/404 in front of display wall302/402 and use the matte to remove display wall 302/402 from the liveaction scene in the image data. Thereafter, the image processor mayreplace background pixels for display wall 302/402 with a rendering ofthe precursor image as it would appear in 3D and/or as a part of thereal-world environment for the live action scene.

Thus, the image matte allows for background adjustment and/orreplacement of display wall pixels that specifically accounts forprecursor metadata of the precursor image on the display wall. Further,the background pixel adjustment or replacement may utilize camerapositions, camera angles, and/or pixel color values to determinereplacement pixels and/or adjust the precursor image. In someembodiments, replacement of pixels for display wall 302/402 maycorrespond to adjusting or replacing captured pixels withstereo-displaced pixels. For example, an image processor may computewhere the camera would be in the precursor scene for the precursor imagebased on a position and/or angle of the camera(s) and display wall302/402. Thus, the background pixels may be generated using thisinformation and the image matte may be used to replace the backgroundpixels for display wall 302/402 with the stereo-displaced pixels. Thismay be done in real-time to adjust the precursor image on display wall302/402 during capture of the live action scene, or during a laterpost-processing operation. Different replacement images may also begenerated for different cameras of a stereoscopic camera set, such ashero or non-hero cameras. The distance, depth, and/or ranginginformation between the camera(s) and display wall 302/402 may be knownfrom position data and/or measurements or may be determined by one ormore depth sensors.

Pixel adjustments, changes, or warping may also be determined and usedfor adjusting pixels with the image matte in additional embodiments.Pixel adjustment may include moving the background pixels of the 2Ddisplay wall in one or more direction and/or warping the pixels so thatthe pixels are adjusted to one or more different locations within theimage data. Adjusting pixels may also include changing data values ofpixels (e.g., color, brightness, luminosity, effect, intensity, and/orthe like) instead of or in addition to movement in order to providedifferent pixel outputs for the original background pixels of thedisplay wall. For example, the background pixels may be adjusted usingthe image matte and the precursor metadata to provide higher resolutionpixels than what was initially captured in image data. This may occurwhere the initial precursor image is quickly rendered and/or of lowresolution, but later a higher resolution image may be desirable, or ifstage, camera, and/or environmental effects and properties cause thedisplay wall to be captured in a lower quality. For example, this mayoccur where camera distance, lens effect, focus, or the like causes thecaptured image data to have lower quality resolution of the precursorimage. Thus, this adjustment by changing or warping pixels may occur inpost-processing, but may also occur in real-time where an initialprecursor image is of lower resolution in order to calibrate camerasand/or precursor metadata.

In some embodiments, the replacement of the background pixels may alsobe performed to provide a fast and/or real-time replacement of onequality, which may be done with the precursor metadata directly ondisplay wall 302/402 or in real-time recorded and viewed image data,while another replacement may be done slower but higher quality at alater time. Additionally, some embodiments may allow the replacement ofbackground pixels to add a visual or photographic effect in a displaywall precursor image or a real-world captured image, such as blurring,color correction, fog or other additional environmental effects, and thelike. This may not be applied to the captured light from the displaywall but is instead applied to the precursor metadata so that it isapplied to the scene description in the precursor metadata. This allowsthe effect to be rendered and substituted for the captured version ofdisplay wall 302/402 in the live action scene.

FIG. 5 is a flowchart of an exemplary method as might be performed by acomputing system when generating an image matte for stereoscopic imagedata captured by stereoscopic cameras of a 3D object and a 2D displaywall, in an embodiment. Note that one or more steps, processes, andmethods described herein of flowchart 500 may be omitted, performed in adifferent sequence, or combined as desired or appropriate.

In step 502 of flowchart 500, stereoscopic image data is received. Thestereoscopic image data may correspond to image data captured by two ormore cameras and include a live action scene where a live actor may beengaging in a performance in 3D. Further, the live action scene mayinclude a display wall that may be generally planar or curved and maypresent a 2D precursor image that corresponds to a 3D scene, character,and/or object that is presented and/or projected in the live actionscene via the display wall. In some embodiments, more generally otherimage data may be received, such as image data from one or more camerasthat need not be stereoscopic image data. For example, other image datamay include background display walls that may be similarly processed asdiscussed herein to determine an image matte and/or adjust pixelscorresponding to the display walls using the image mat.

In step 504, metadata for a precursor image and the display wall isdetermined. The display wall may include pixels that correspond to thedisplayed precursor image, which may be displayed using precursormetadata from a renderer of the precursor image and display wallmetadata for a geometry and other attributes of a display wall. Theprecursor metadata may include a scene description of the displayedscene for the display wall and therefore may include pixel display datafor pixels displayed on the display wall. Additionally, captured imagedata further includes pixels, which may correspond to both the liveactor and the display wall in the image data. In this regard, whenreferring to background pixels of the display wall, the backgroundpixels may be replaced by replacement pixels on the display walldirectly (e.g., while capturing the live action scene with the displaywall) or may be replaced later in the image data during post-processing.In other embodiments, pixel values and/or data for the background pixelsmay instead be adjusted, warped, and/or changed based on desired changesto those background pixels on the display wall directly or in thecorresponding image data. The display wall metadata may include geometryand aspects of the display wall needed for rendering the precursor image

In step 506, a portion of the stereoscopic image data having a liveactor is determined. In step 508, a portion of the stereoscopic imagedata having the display wall is determined. These may be two or moreseparate portions where pixel values are identified as corresponding tothe live actor, the display wall, or one or more other elements and/orpixel identifiers. In some embodiments, steps 506 and/or 508 may occurin a different order or at the same time, such as when the portion ofthe stereoscopic image data belonging to the display wall is determinedfirst or along with the portion belonging to the actor, respectively. Insuch embodiments, the display wall and/or precursor metadata may beused, at least in part, to identify the respective portions of the imagedata belonging to the live actor and the display wall. In otherembodiments, different stage elements may also or instead be present inthe stereoscopic image data, such as 3D objects, characters, orcreatures different from a live actor.

When determining the portions in steps 506 and 508, one or morecomputations may be executed using the display wall and/or precursormetadata with the stereoscopic image data to identify the respectiveportions of the live actor and the display wall (and/or other objectsand/or displays that may be in the live action scene and/or precursorimages). Computation of the portions belonging to the live actor or thedisplay wall may be performed by analyzing the display wall and/orprecursor metadata to determine pixel values for the rendered precursorimage and computing which pixel values in the captured image data matchor are similar, within a similarity pixel value tolerance, similarity,or range, to those pixel values from the precursor metadata. Matchingmay be based on color values, brightness and/or luminosity, and/orrelative location in the captured image data.

Further, a placement of the display wall relative to each cameracapturing the image data may be used to determine foreground pixels ofthe live actor or other objects in the live action scene and backgroundpixels of the display wall that are a known or measured distance, depth,or range from each camera. The distance of foreground pixels may beidentified by those not matching the distance of the display wall fromthe cameras based on relative placements of the cameras and the displaywall. In contrast, those that match the expected display wall distancemay be designated as belonging to the display wall. Distance between thecameras and the display wall may be determined using one or moredistance, depth, or ranging sensors and/or optical capture devices.

In step 510, an image matte for the portions of the stereoscopic imagedata is generated. The image matte may be generated using the two ormore portions that identify the live actor, the display wall, anotherobject or element, or have another identifier (e.g., a pixel valueshared by the live actor and display wall or a pixel value that an imageprocessor is unsure of whether the corresponding pixel belongs to thelive actor or display wall). In some embodiments, the image matte maycorrespond to a binary image matte having one of two possible pixelvalues or identifiers for each of the live actor or the display wall.The image matte may therefore correspond to an alpha channel image thatmay allow for adjustments of portions of the stereoscopic image data.

In other embodiments, the image matte may correspond to a junk imagematte or an n-ary image mat. With a junk image mat, pixels may beidentified as those belonging to a particular object, character, or thelike, such as the display wall or live actor, and those pixels that donot belong to this object. Further a categorization for the junk imagematte may include those that are uncertain to belong to a specific groupor pixel value. An n-ary image matte may also be used to identify pixelvalues that correspond to both the live actor and the display wall. Theimage matte may be stored as a pixel array and allow for modification ofbackground pixels of the display wall.

FIG. 6 is a flowchart of an exemplary method as might be performed by acomputing system when rendering and/or adjusting pixels for use withimage data of live action scenes having a 3D object and a 2D displaywall, in an embodiment. Note that one or more steps, processes, andmethods described herein of flowchart 600 may be omitted, performed in adifferent sequence, or combined as desired or appropriate.

In step 602 of flowchart 600, stereoscopic image data is received. Thestereoscopic image data may correspond to the same or similar image datacaptured in step 502 of FIG. 5 . However, in step 602, the stereoscopicimage data is received after an image matte is determined and generatedin order for the image matte to be applied to the image data andbackground pixels in the image data and/or on the display wall to bereplaced, moved, warped, or otherwise changed. In this regard, the imagedata may include a live action scene captured by one or more cameras,such as two stereoscopic cameras when capturing a scene from differentangles.

In step 604, metadata for a precursor image and a display wall isdetermined. As with step 504 of FIG. 5 , precursor metadata maysimilarly correspond to data that may be used to render and/or output aprecursor image on the display wall, which may include pixel displaydata for display wall pixels. Further, the precursor metadata mayinclude a scene description for the precursor image and otherinformation that may designate the output and/or visualization of theprecursor image for the display wall and for the live action scene.Similarly, display wall metadata may be used to determine a geometryand/or placement of the display wall.

In step 606, the image matte for portions of the stereoscopic image datais accessed. The image matte may correspond to the one generated as theoutput of flowchart 500 from FIG. 5 , such as the binary, junk, or n-aryimage mat. In this regard, the image matte may be used to identify thepixels in the image data that correspond to the background pixels forthe display wall, as well as foreground pixels, actor, object, or thelike for foreground objects in the live action scene. Thus, the imagematte may be used to specifically identify the pixel values for thedisplay data so that the precursor image on the display wall may bere-rendered and/or adjusted for output for the display wall (e.g., inreal-time when filming or capturing the live action scene and/or duringpost-processing). The image matte may also be used during real-time orpost-processing of image data to adjust and/or re-render the precursorimage in the captured image data and provide new background pixels insuch image data. This may be performed to correct for camera-inducederror or effects, resolution, and the like, provide a special image orreal-world effect, or to adjust or replace with stereo-displaced pixelsthat may cause the precursor image to appear as a part of the 3Denvironment for the live action scene.

In step 608, the precursor image on the display wall is adjusted orre-rendered using the image mat. For example, when capturingstereoscopic or other image data from one or more cameras of a liveaction scene having a display wall with foreground actors or objects,different camera angles and/or placements may cause different capturingof the pixels, including different background pixels of the display wallthat may be blocked or visible. Additionally, the background pixels'color, luminosity, or intensity, and the like may be different based oncamera settings, scene lighting or effects in the 3D real-worldenvironment, and the like. This may be caused by the placements or theangles of the cameras but may also be caused due to lighting in the liveaction scene, camera construction and/or engineering, camera lens, addedor modified visual or special effects, actor lighting or costume, andother changes caused by elements of the display wall and/or liveenvironment. This may include modifying properties, data or displayvalues of pixels, and the like for different effects and visuals.

When adjusting or re-rendering the precursor image, the backgroundpixels may be moved or warped in one or more sets of image data and/orreplacement pixels may be generated for the display wall when capturingimage data in real-time or capturing additional image data of the liveaction scene at a later time. The adjustments or replacement pixels maycorrespond to moving or warping the original display wall pixels in thebackground of the stereoscopic image data so that the image data hasmoved or warped pixels, or may include generating replacement pixels tochange the background image, which may cause the background display wallpixels to appear different and/or stereoscopically adjusted. Forexample, when capturing the 2D display wall stereoscopically using twoor more cameras, the different placements of the cameras and/or displaywall may cause different background pixels of the display wall to becaptured. Determination of the different pixels may be based on theprecursor metadata, the display wall metadata, and/or the image matte.

Once differences between background pixels in different image data setsare determined, compensation may be provided for placements ofbackground pixels, such as by moving or warping those pixels, so thatthe different image data sets may be joined or formed to be stereoscopicor other image effects may be created. For example, an average or othermovement distance between different distances or placements betweendifferent background pixels in the different image data may betdetermined and the background pixels for the display wall may be movedor warped in the image data. This may allow for a stereoscopic effect tobe applied to the image data. Further, other movements or warps may beapplied in order to provide different effects and/or cause the displaywall to appear differently in the image data.

In various embodiments, the precursor image may not appear as renderedand designated by the precursor metadata. Thus, re-rendering the imagemay include determining the background pixels belonging to the displaywall and generating replacement pixels and/or adjustingdisplayed/captured pixels to adjusted pixel values based on the imagemat, the captured image data, background wall metadata, and/or theprecursor metadata. The changes, warping properties, and/or adjustmentsmay be generated to provide different colors and/or pixel values (e.g.,brightness, intensity, luminosity, etc.), such as where the color orpixel values in the image data do not match the desired pixel values forthe precursor metadata. The adjustments or replacement pixels may alsobe generated for the display wall when capturing the live action sceneto provide better resolution of the precursor image and/or add or modifya visual or special effect in a precursor and/or real-world portion ofimages (e.g., pixels for an image or real-world effect. In furtherembodiments, the adjustments or replacement pixels may correspond tostereo-displaced pixels that may cause the precursor image on thedisplay wall to appear as a 3D environment, character, creature, and/orobject that is part of the live environment for the live actor in thelive action scene. Additionally, the adjustments or replacement pixelsmay be generated for the image data so that in post-processing,background pixels in the image data may be adjusted and/or replaced.This may be done for the aforementioned effects and/or alterations ofthe precursor image, such as to provide different pixel values, providebetter or different resolution, and/or add stereo-displaced pixels thatcause the precursor image to appear in 3D as part of the live actionscene's 3D environment.

FIG. 7 illustrates a system 700 for processing precursor imagesdisplayed on a 2D display wall during capture of a live action scene, inan embodiment. System 700 includes a precursor image dataset 702, adisplay wall processing unit 706, a renderer 718, a UI 720, andprecursor image metadata 722.

A user 740 may interact with the UI 720 to define one or more precursorimages for display on a display wall in a live action scene with one ormore live actors or other real-world objects in the live action scene. Adisplay wall may therefore correspond to a 2D image display and/oroutput component that is in a 3D scene. Precursor image metadata 722 mayindicate, for example, the criteria for generation and/or display of oneor more precursor images, which is further captured in the live actionscene. Precursor image dataset 702 may store UI data used to present oneor more precursor images that are input, adjusted, and/or generated viaUI 720, which provides an improved UI for precursor image display.Precursor image dataset 702 may also include data and metadata used torender the precursor images. Precursor image dataset 702 may be loadedwith data from a source of an animation and/or live captured scene, suchas images, videos, and the like that is to be output on a display wallcaptured in a live action scene. Display wall processing unit 706 mayutilize the methods and processes described herein to take precursorimage metadata 722 and generate and/or display the precursor image(s) onthe display wall. The display wall processing unit 706 may generateand/or adjust the precursor image during and/or after capture of thelive action scene stereoscopically so that the precursor image may beadjusted, as described herein.

Display wall processing unit 706 includes a processor 710 that executesprogram code 712 to generate, adjust, and/or display one or moreprecursor images based on precursor image metadata 722, such as initialimage data and metadata contained in precursor data 714. Display wallprocessing unit 706 may generate adjusted display wall data 716 fromsimplified UI inputs using UI 720, which may be the corresponding outputof a precursor image on a display wall and/or in image data of thecaptured display wall. Display wall processing unit 706 may furtherstore precursor image adjustments 708 to precursor image dataset 702 sothat the corresponding data structures may later be used as guidefeathers during groom generation. For example, display wall processingunit 706 may initiate the process by taking precursor image metadata 722with initial precursor data 714 and generating a digital representationof a feather having adjusted display wall data 716. Display wallprocessing unit 706 may then output precursor image adjustments 708,which may be included with precursor image and metadata specifications704 stored by precursor image dataset 702. Display wall processing unit706 may then move to the next precursor image designated by user 740 andfurther generate and/or adjust precursor images. The resulting generatedand/or adjusted precursor images may be rendered by renderer 718 and/oroutput to user 740 to inspect the results.

Note that, in the context of describing disclosed embodiments, unlessotherwise specified, use of expressions regarding executableinstructions (also referred to as code, applications, agents, etc.)performing operations that “instructions” do not ordinarily performunaided (e.g., transmission of data, calculations, etc.) denotes thatthe instructions are being executed by a machine, thereby causing themachine to perform the specified operations.

As one skilled in the art will appreciate in light of this disclosure,certain embodiments may be capable of achieving certain advantages,including some or all of the following: (1) Techniques described andsuggested in the present disclosure improve the field of computing,especially the field of digital animation, by improving the computationtime and visualization of precursor images displayed on display walls.(2) Additionally, techniques described and suggested in the presentdisclosure improve the efficiency of computing systems by, since thecomputation time to calculate precursor image adjustments is reduced, tocompute and render more complex and realistic models in digitalanimation and/or live action sequences. (3) Moreover, techniquesdescribed and suggested in the present disclosure are necessarily rootedin computer technology in order to overcome problems specificallyarising with how to generate and/or adjust precursor images on displaywalls and/or in image data of captured display walls within thecomputational and time constraints of producing a digital animationproduct.

FIG. 8 illustrates an example visual content generation system 800 asmight be used to generate imagery in the form of still images and/orvideo sequences of images. Visual content generation system 800 mightgenerate imagery of live action scenes, computer generated scenes, or acombination thereof. In a practical system, users are provided withtools that allow them to specify, at high levels and low levels wherenecessary, what is to go into that imagery. For example, a user might bean animation artist and might use visual content generation system 800to capture interaction between two human actors performing live on asound stage and replace one of the human actors with acomputer-generated anthropomorphic non-human being that behaves in waysthat mimic the replaced human actor's movements and mannerisms, and thenadd in a third computer-generated character and background stageelements that are computer-generated, all in order to tell a desiredstory or generate desired imagery.

Still images that are output by visual content generation system 800might be represented in computer memory as pixel arrays, such as atwo-dimensional array of pixel color values, each associated with apixel having a position in a two-dimensional image array. Pixel colorvalues might be represented by three or more (or fewer) color values perpixel, such as a red value, a green value, and a blue value (e.g., inRGB format). Dimensions of such a two-dimensional array of pixel colorvalues might correspond to a preferred and/or standard display scheme,such as 1920-pixel columns by 1280-pixel rows or 4096-pixel columns by2160-pixel rows, or some other resolution Images might or might not bestored in a certain structured format, but either way, a desired imagemay be represented as a two-dimensional array of pixel color values. Inanother variation, images are represented by a pair of stereo images forthree-dimensional presentations and in other variations, an imageoutput, or a portion thereof, might represent three-dimensional imageryinstead of just two-dimensional views. In yet other embodiments, pixelvalues are data structures, and a pixel value can be associated with apixel and can be a scalar value, a vector, or another data structureassociated with a corresponding pixel. That pixel value might includecolor values, or not, and might include depth values, alpha values,weight values, object identifiers or other pixel value components.

A stored video sequence might include a plurality of images such as thestill images described above, but where each image of the plurality ofimages has a place in a timing sequence and the stored video sequence isarranged so that when each image is displayed in order, at a timeindicated by the timing sequence, the display presents what appears tobe moving and/or changing imagery. In one representation, each image ofthe plurality of images is a video frame having a specified frame numberthat corresponds to an amount of time that would elapse from when avideo sequence begins playing until that specified frame is displayed. Aframe rate might be used to describe how many frames of the stored videosequence are displayed per unit time. Example video sequences mightinclude 24 frames per second (24 FPS), 50 FPS, 140 FPS, or other framerates. In some embodiments, frames are interlaced or otherwise presentedfor display, but for clarity of description, in some examples, it isassumed that a video frame has one specified display time, but othervariations might be contemplated.

One method of creating a video sequence is to simply use a video camerato record a live action scene, i.e., events that physically occur andcan be recorded by a video camera. The events being recorded can beevents to be interpreted as viewed (such as seeing two human actors talkto each other) and/or can include events to be interpreted differentlydue to clever camera operations (such as moving actors about a stage tomake one appear larger than the other despite the actors actually beingof similar build or using miniature objects with other miniature objectsso as to be interpreted as a scene containing life-sized objects).

Creating video sequences for story-telling or other purposes often callsfor scenes that cannot be created with live actors, such as a talkingtree, an anthropomorphic object, space battles, and the like. Such videosequences might be generated computationally rather than capturing lightfrom live scenes. In some instances, an entirety of a video sequencemight be generated computationally, as in the case of acomputer-animated feature film. In some video sequences, it is desirableto have some computer-generated imagery and some live action, perhapswith some careful merging of the two.

While computer-generated imagery might be creatable by manuallyspecifying each color value for each pixel in each frame, this is likelytoo tedious to be practical. As a result, a creator uses various toolsto specify the imagery at a higher level. As an example, an artist mightspecify the positions in a scene space, such as a three-dimensionalcoordinate system, of objects and/or lighting, as well as a cameraviewpoint, and a camera view plane. From that, a rendering engine couldtake all of those as inputs, and compute each of the pixel color valuesin each of the frames. In another example, an artist specifies positionand movement of an articulated object having some specified texturerather than specifying the color of each pixel representing thatarticulated object in each frame.

In a specific example, a rendering engine performs ray tracing wherein apixel color value is determined by computing which objects lie along aray traced in the scene space from the camera viewpoint through a pointor portion of the camera view plane that corresponds to that pixel. Forexample, a camera view plane might be represented as a rectangle havinga position in the scene space that is divided into a grid correspondingto the pixels of the ultimate image to be generated, and if a raydefined by the camera viewpoint in the scene space and a given pixel inthat grid first intersects a solid, opaque, blue object, that givenpixel is assigned the color blue. Of course, for moderncomputer-generated imagery, determining pixel colors—and therebygenerating imagery—can be more complicated, as there are lightingissues, reflections, interpolations, and other considerations.

As illustrated in FIG. 8 , a live action capture system 802 captures alive scene that plays out on a stage 804. Live action capture system 802is described herein in greater detail, but might include computerprocessing capabilities, image processing capabilities, one or moreprocessors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown.

In a specific live action capture system, cameras 806(1) and 806(2)capture the scene, while in some systems, there might be other sensor(s)808 that capture information from the live scene (e.g., infraredcameras, infrared sensors, motion capture (“mo-cap”) detectors, etc.).On stage 804, there might be human actors, animal actors, inanimateobjects, background objects, and possibly an object such as a greenscreen 810 that is designed to be captured in a live scene recording insuch a way that it is easily overlaid with computer-generated imagery.Stage 804 might also contain objects that serve as fiducials, such asfiducials 812(1)-(3), that might be used post-capture to determine wherean object was during capture. A live action scene might be illuminatedby one or more lights, such as an overhead light 814.

During or following the capture of a live action scene, live actioncapture system 802 might output live action footage to a live actionfootage storage 820. A live action processing system 822 might processlive action footage to generate data about that live action footage andstore that data into a live action metadata storage 824. Live actionprocessing system 822 might include computer processing capabilities,image processing capabilities, one or more processors, program codestorage for storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Live action processing system 822 might processlive action footage to determine boundaries of objects in a frame ormultiple frames, determine locations of objects in a live action scene,where a camera was relative to some action, distances between movingobjects and fiducials, etc. Where elements have sensors attached to themor are detected, the metadata might include location, color, andintensity of overhead light 814, as that might be useful inpost-processing to match computer-generated lighting on objects that arecomputer-generated and overlaid on the live action footage. Live actionprocessing system 822 might operate autonomously, perhaps based onpredetermined program instructions, to generate and output the liveaction metadata upon receiving and inputting the live action footage.The live action footage can be camera-captured data as well as data fromother sensors.

An animation creation system 830 is another part of visual contentgeneration system 800. Animation creation system 830 might includecomputer processing capabilities, image processing capabilities, one ormore processors, program code storage for storing program instructionsexecutable by the one or more processors, as well as user input devicesand user output devices, not all of which are shown. Animation creationsystem 830 might be used by animation artists, managers, and others tospecify details, perhaps programmatically and/or interactively, ofimagery to be generated. From user input and data from a database orother data source, indicated as a data store 832, animation creationsystem 830 might generate and output data representing objects (e.g., ahorse, a human, a ball, a teapot, a cloud, a light source, a texture,etc.) to an object storage 834, generate and output data representing ascene into a scene description storage 836, and/or generate and outputdata representing animation sequences to an animation sequence storage838.

Scene data might indicate locations of objects and other visualelements, values of their parameters, lighting, camera location, cameraview plane, and other details that a rendering engine 850 might use torender CGI imagery. For example, scene data might include the locationsof several articulated characters, background objects, lighting, etc.specified in a two-dimensional space, three-dimensional space, or otherdimensional space (such as a 2.5-dimensional space, three-quarterdimensions, pseudo-3D spaces, etc.) along with locations of a cameraviewpoint and view place from which to render imagery. For example,scene data might indicate that there is to be a red, fuzzy, talking dogin the right half of a video and a stationary tree in the left half ofthe video, all illuminated by a bright point light source that is aboveand behind the camera viewpoint. In some cases, the camera viewpoint isnot explicit, but can be determined from a viewing frustum. In the caseof imagery that is to be rendered to a rectangular view, the frustumwould be a truncated pyramid. Other shapes for a rendered view arepossible and the camera view plane could be different for differentshapes.

Animation creation system 830 might be interactive, allowing a user toread in animation sequences, scene descriptions, object details, etc.and edit those, possibly returning them to storage to update or replaceexisting data. As an example, an operator might read in objects fromobject storage into a baking processor 842 that would transform thoseobjects into simpler forms and return those to object storage 834 as newor different objects. For example, an operator might read in an objectthat has dozens of specified parameters (movable joints, color options,textures, etc.), select some values for those parameters and then save abaked object that is a simplified object with now fixed values for thoseparameters.

Rather than requiring user specification of each detail of a scene, datafrom data store 832 might be used to drive object presentation. Forexample, if an artist is creating an animation of a spaceship passingover the surface of the Earth, instead of manually drawing or specifyinga coastline, the artist might specify that animation creation system 830is to read data from data store 832 in a file containing coordinates ofEarth coastlines and generate background elements of a scene using thatcoastline data.

Animation sequence data might be in the form of time series of data forcontrol points of an object that has attributes that are controllable.For example, an object might be a humanoid character with limbs andjoints that are movable in manners similar to typical human movements.An artist can specify an animation sequence at a high level, such as“the left hand moves from location (X1, Y1, Z1) to (X2, Y2, Z2) overtime T1 to T2”, at a lower level (e.g., “move the elbow joint 2.5degrees per frame”) or even at a very high level (e.g., “character Ashould move, consistent with the laws of physics that are given for thisscene, from point P1 to point P2 along a specified path”).

Animation sequences in an animated scene might be specified by whathappens in a live action scene. An animation driver generator 844 mightread in live action metadata, such as data representing movements andpositions of body parts of a live actor during a live action sceneAnimation driver generator 844 might generate corresponding animationparameters to be stored in animation sequence storage 838 for use inanimating a CGI object. This can be useful where a live action scene ofa human actor is captured while wearing mo-cap fiducials (e.g.,high-contrast markers outside actor clothing, high-visibility paint onactor skin, face, etc.) and the movement of those fiducials isdetermined by live action processing system 822. Animation drivergenerator 844 might convert that movement data into specifications ofhow joints of an articulated CGI character are to move over time.

A rendering engine 850 can read in animation sequences, scenedescriptions, and object details, as well as rendering engine controlinputs, such as a resolution selection and a set of renderingparameters. Resolution selection might be useful for an operator tocontrol a trade-off between speed of rendering and clarity of detail, asspeed might be more important than clarity for a movie maker to testsome interaction or direction, while clarity might be more importantthan speed for a movie maker to generate data that will be used forfinal prints of feature films to be distributed. Rendering engine 850might include computer processing capabilities, image processingcapabilities, one or more processors, program code storage for storingprogram instructions executable by the one or more processors, as wellas user input devices and user output devices, not all of which areshown.

Visual content generation system 800 can also include a merging system860 that merges live footage with animated content. The live footagemight be obtained and input by reading from live action footage storage820 to obtain live action footage, by reading from live action metadatastorage 824 to obtain details such as presumed segmentation in capturedimages segmenting objects in a live action scene from their background(perhaps aided by the fact that green screen 810 was part of the liveaction scene), and by obtaining CGI imagery from rendering engine 850.

A merging system 860 might also read data from rulesets formerging/combining storage 862. A very simple example of a rule in aruleset might be “obtain a full image including a two-dimensional pixelarray from live footage, obtain a full image including a two-dimensionalpixel array from rendering engine 850, and output an image where eachpixel is a corresponding pixel from rendering engine 850 when thecorresponding pixel in the live footage is a specific color of green,otherwise output a pixel value from the corresponding pixel in the livefootage.”

Merging system 860 might include computer processing capabilities, imageprocessing capabilities, one or more processors, program code storagefor storing program instructions executable by the one or moreprocessors, as well as user input devices and user output devices, notall of which are shown. Merging system 860 might operate autonomously,following programming instructions, or might have a user interface orprogrammatic interface over which an operator can control a mergingprocess. In some embodiments, an operator can specify parameter valuesto use in a merging process and/or might specify specific tweaks to bemade to an output of merging system 860, such as modifying boundaries ofsegmented objects, inserting blurs to smooth out imperfections, oradding other effects. Based on its inputs, merging system 860 can outputan image to be stored in a static image storage 870 and/or a sequence ofimages in the form of video to be stored in an animated/combined videostorage 872.

Thus, as described, visual content generation system 800 can be used togenerate video that combines live action with computer-generatedanimation using various components and tools, some of which aredescribed in more detail herein. While visual content generation system800 might be useful for such combinations, with suitable settings, itcan be used for outputting entirely live action footage or entirely CGIsequences. The code may also be provided and/or carried by a transitorycomputer readable medium, e.g., a transmission medium such as in theform of a signal transmitted over a network.

According to one embodiment, the techniques described herein areimplemented by one or more generalized computing systems programmed toperform the techniques pursuant to program instructions in firmware,memory, other storage, or a combination. Special-purpose computingdevices may be used, such as desktop computer systems, portable computersystems, handheld devices, networking devices or any other device thatincorporates hard-wired and/or program logic to implement thetechniques.

One embodiment might include a carrier medium carrying image data orother data having details generated using the methods described herein.The carrier medium can comprise any medium suitable for carrying theimage data or other data, including a storage medium, e.g., solid-statememory, an optical disk or a magnetic disk, or a transient medium, e.g.,a signal carrying the image data such as a signal transmitted over anetwork, a digital signal, a radio frequency signal, an acoustic signal,an optical signal or an electrical signal.

FIG. 9 is a block diagram that illustrates a computer system 900 uponwhich the computer systems of the systems described herein and/or visualcontent generation system 800 (see FIG. 8 ) may be implemented. Computersystem 900 includes a bus 902 or other communication mechanism forcommunicating information, and a processor 904 coupled with bus 902 forprocessing information. Processor 904 may be, for example, ageneral-purpose microprocessor.

Computer system 900 also includes a main memory 906, such as arandom-access memory (RAM) or other dynamic storage device, coupled tobus 902 for storing information and instructions to be executed byprocessor 904. Main memory 906 may also be used for storing temporaryvariables or other intermediate information during execution ofinstructions to be executed by processor 904. Such instructions, whenstored in non-transitory storage media accessible to processor 904,render computer system 900 into a special-purpose machine that iscustomized to perform the operations specified in the instructions.

Computer system 900 further includes a read only memory (ROM) 908 orother static storage device coupled to bus 902 for storing staticinformation and instructions for processor 904. A storage device 910,such as a magnetic disk or optical disk, is provided and coupled to bus902 for storing information and instructions.

Computer system 900 may be coupled via bus 902 to a display 912, such asa computer monitor, for displaying information to a computer user. Aninput device 914, including alphanumeric and other keys, is coupled tobus 902 for communicating information and command selections toprocessor 904. Another type of user input device is a cursor control916, such as a mouse, a trackball, or cursor direction keys forcommunicating direction information and command selections to processor904 and for controlling cursor movement on display 912. This inputdevice typically has two degrees of freedom in two axes, a first axis(e.g., x) and a second axis (e.g., y), that allows the device to specifypositions in a plane.

Computer system 900 may implement the techniques described herein usingcustomized hard-wired logic, one or more ASICs or FPGAs, firmware and/orprogram logic which in combination with the computer system causes orprograms computer system 900 to be a special-purpose machine. Accordingto one embodiment, the techniques herein are performed by computersystem 900 in response to processor 904 executing one or more sequencesof one or more instructions contained in main memory 906. Suchinstructions may be read into main memory 906 from another storagemedium, such as storage device 910. Execution of the sequences ofinstructions contained in main memory 906 causes processor 904 toperform the process steps described herein. In alternative embodiments,hard-wired circuitry may be used in place of or in combination withsoftware instructions.

The term “storage media” as used herein refers to any non-transitorymedia that store data and/or instructions that cause a machine tooperation in a specific fashion. Such storage media may includenon-volatile media and/or volatile media. Non-volatile media includes,for example, optical or magnetic disks, such as storage device 910.Volatile media includes dynamic memory, such as main memory 906. Commonforms of storage media include, for example, a floppy disk, a flexibledisk, hard disk, solid state drive, magnetic tape, or any other magneticdata storage medium, a CD-ROM, any other optical data storage medium,any physical medium with patterns of holes, a RAM, a PROM, an EPROM, aFLASH-EPROM, NVRAM, any other memory chip or cartridge.

Storage media is distinct from but may be used in conjunction withtransmission media. Transmission media participates in transferringinformation between storage media. For example, transmission mediaincludes coaxial cables, copper wire, and fiber optics, including thewires that include bus 902. Transmission media can also take the form ofacoustic or light waves, such as those generated during radio-wave andinfra-red data communications.

Various forms of media may be involved in carrying one or more sequencesof one or more instructions to processor 904 for execution. For example,the instructions may initially be carried on a magnetic disk orsolid-state drive of a remote computer. The remote computer can load theinstructions into its dynamic memory and send the instructions over anetwork connection. A modem or network interface local to computersystem 900 can receive the data. Bus 902 carries the data to main memory906, from which processor 904 retrieves and executes the instructions.The instructions received by main memory 906 may optionally be stored onstorage device 910 either before or after execution by processor 904.

Computer system 900 also includes a communication interface 918 coupledto bus 902. Communication interface 918 provides a two-way datacommunication coupling to a network link 920 that is connected to alocal network 922. For example, communication interface 918 may be anetwork card, a modem, a cable modem, or a satellite modem to provide adata communication connection to a corresponding type of telephone lineor communications line. Wireless links may also be implemented. In anysuch implementation, communication interface 918 sends and receiveselectrical, electromagnetic, or optical signals that carry digital datastreams representing various types of information.

Network link 920 typically provides data communication through one ormore networks to other data devices. For example, network link 920 mayprovide a connection through local network 922 to a host computer 924 orto data equipment operated by an Internet Service Provider (ISP) 926.ISP 926 in turn provides data communication services through theworld-wide packet data communication network now commonly referred to asthe “Internet” 928. Local network 922 and Internet 928 both useelectrical, electromagnetic, or optical signals that carry digital datastreams. The signals through the various networks and the signals onnetwork link 920 and through communication interface 918, which carrythe digital data to and from computer system 900, are example forms oftransmission media.

Computer system 900 can send messages and receive data, includingprogram code, through the network(s), network link 920, andcommunication interface 918. In the Internet example, a server 930 mighttransmit a requested code for an application program through theInternet 928, ISP 926, local network 922, and communication interface918. The received code may be executed by processor 904 as it isreceived, and/or stored in storage device 910, or other non-volatilestorage for later execution.

Operations of processes described herein can be performed in anysuitable order unless otherwise indicated herein or otherwise clearlycontradicted by context. Processes described herein (or variationsand/or combinations thereof) may be performed under the control of oneor more computer systems configured with executable instructions and maybe implemented as code (e.g., executable instructions, one or morecomputer programs or one or more applications) executing collectively onone or more processors, by hardware or combinations thereof. The codemay be stored on a computer-readable storage medium, for example, in theform of a computer program comprising a plurality of instructionsexecutable by one or more processors. The computer-readable storagemedium may be non-transitory. The code may also be provided carried by atransitory computer readable medium e.g., a transmission medium such asin the form of a signal transmitted over a network.

Conjunctive language, such as phrases of the form “at least one of A, B,and C,” or “at least one of A, B and C,” unless specifically statedotherwise or otherwise clearly contradicted by context, is otherwiseunderstood with the context as used in general to present that an item,term, etc., may be either A or B or C, or any nonempty subset of the setof A and B and C. For instance, in the illustrative example of a sethaving three members, the conjunctive phrases “at least one of A, B, andC” and “at least one of A, B and C” refer to any of the following sets:{A}, {B}, {C}, {A, B}, {A, C}, {B, C}, {A, B, C}. Thus, such conjunctivelanguage is not generally intended to imply that certain embodimentsrequire at least one of A, at least one of B and at least one of C eachto be present.

The use of examples, or exemplary language (e.g., “such as”) providedherein, is intended merely to better illuminate embodiments of theinvention and does not pose a limitation on the scope of the inventionunless otherwise claimed. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the invention.

In the foregoing specification, embodiments of the invention have beendescribed with reference to numerous specific details that may vary fromimplementation to implementation. The specification and drawings are,accordingly, to be regarded in an illustrative rather than a restrictivesense. The sole and exclusive indicator of the scope of the invention,and what is intended by the applicants to be the scope of the invention,is the literal and equivalent scope of the set of claims that issue fromthis application, in the specific form in which such claims issue,including any subsequent correction.

Further embodiments can be envisioned to one of ordinary skill in theart after reading this disclosure. In other embodiments, combinations orsub-combinations of the above-disclosed invention can be advantageouslymade. The example arrangements of components are shown for purposes ofillustration and combinations, additions, re-arrangements, and the likeare contemplated in alternative embodiments of the present invention.Thus, while the invention has been described with respect to exemplaryembodiments, one skilled in the art will recognize that numerousmodifications are possible.

For example, the processes described herein may be implemented usinghardware components, software components, and/or any combinationthereof. The specification and drawings are, accordingly, to be regardedin an illustrative rather than a restrictive sense. It will, however, beevident that various modifications and changes may be made thereuntowithout departing from the broader spirit and scope of the invention asset forth in the claims and that the invention is intended to cover allmodifications and equivalents within the scope of the following claims.

All references, including publications, patent applications, andpatents, cited herein are hereby incorporated by reference to the sameextent as if each reference were individually and specifically indicatedto be incorporated by reference and were set forth in its entiretyherein.

What is claimed is:
 1. A computer-implemented method for processing, inan image processing system, a captured scene captured of a live actionscene while a display wall is positioned to be part of the live actionscene, wherein the display wall comprises one or more structures capableof displaying imagery, the method comprising: receiving image data ofthe live action scene having a live actor and the display walldisplaying a first rendering of a precursor image, wherein the imagedata is captured by at least one camera in at least one placementrelative to the live actor and the display wall; determining precursormetadata for the precursor image displayed on the display wall anddisplay wall metadata for the display wall, wherein the precursormetadata comprises pixel display data for display wall pixels of thedisplay wall, and wherein the display wall metadata comprises geometrydata for a display wall position relative to the at least one camera;accessing an image matte for the image data, wherein the image matteindicates a first portion associated with the live actor and a secondportion associated with the precursor image on the display wall in thelive action scene; determining pixel display values to add or modify atleast one of an image effect from the display wall pixels of theprecursor image or a visual effect provided in the live action scene,and wherein the pixel display values comprise one or more adjustments topixels in the image data; and adjusting the image data of the capturedscene using the pixel display values and the image matte to add ormodify the at least one of the image effect or the visual effectindependent of rendering of the precursor image on the display wallduring capture of the image data.
 2. The computer-implemented method ofclaim 1, wherein adjusting the image data to add or modify the at leastone of the image effect or the visual effect comprises modifyingbackground pixels of the display wall in the image data based on thepixel display values.
 3. The computer-implemented method of claim 2,wherein the pixel display values are determined at a time aftercapturing the image data to add the at least one of the image effect orthe visual effect to the image data based on the one or more adjustmentsindependent of re-rendering the precursor image on the display wall inthe live action scene or the image data, and wherein adjusting the imagedata to add or modify the at least one of the image effect or the visualeffect is performed at the time after capturing the image data.
 4. Thecomputer-implemented method of claim 2, wherein the pixel display valuesare determined in real-time during a capture of the image data to addthe at least one of the image effect or the visual effect to the imagedata based on the one or more adjustments independent of re-renderingthe precursor image on the display wall in the live action scene or theimage data, and wherein adjusting the image data to add or modify the atleast one of the image effect or the visual effect is performed inreal-time during the capture of the image data.
 5. Thecomputer-implemented method of claim 1, wherein, before determining thepixel display values, the method further comprises: determining one ormore changes to the precursor metadata for the at least one of the imageeffect or the visual effect; and updating the precursor metadata basedon the one or more changes, wherein determining the pixel display valuesuses the updated precursor metadata.
 6. The computer-implemented methodof claim 1, wherein adjusting the image data to add or modify the atleast one of the image effect or the visual effect comprises modifyingforeground pixels corresponding to one or more of the live actor or astage element in the image data based on the pixel display values. 7.The computer-implemented method of claim 1, wherein the at least onecamera comprises two cameras stereoscopically oriented in at least twoplacements and relative to each other when capturing the live actionscene, wherein the image data comprises stereoscopic image data, andwherein determining the pixel display values uses the stereoscopic imagedata.
 8. The computer-implemented method of claim 7, wherein the imagematte comprises a depth determined using the two cameras to one of thelive actor or the display wall, and wherein adjusting the image data toadd or modify the at least one of the image effect or the visual effectuses the depth to change a property of a corresponding effect.
 9. Thecomputer-implemented method of claim 1, wherein one or more of the imageeffect or the visual effect comprise at least one of an environmentaleffect, a lighting effect, an elemental effect, a venue-based effect, ablurring effect, a focus effect, an increased resolution effect, or acolor correction.
 10. The computer-implemented method of claim 1,wherein, before accessing the image matte, the method further comprises:generating the image matte using at least one of the image data, thedisplay wall metadata, or the precursor metadata based on one or moredepths detected of the first portion associated with the live actor andthe second portion associated with the precursor image on the displaywall in the live action scene.
 11. The computer-implemented method ofclaim 1, wherein the precursor image on the display wall comprises acomputer animated background scene having at least one of lightprojected on the live actor or one or more computer animated objects forinteraction with the live actor.
 12. A system comprising: at least oneprocessor, and a storage medium storing instructions, which whenexecuted by the at least one processor, cause the system to implementthe computer-implemented method of claim
 1. 13. A non-transitorycomputer-readable storage medium storing instructions, which whenexecuted by at least one processor of a computer system, causes thecomputer system to carry out the computer-implemented method of claim 1.14. A non-transitory computer-readable medium carrying instructions,which when executed by at least one processor of a computer system,causes the computer system to carry out the computer-implemented methodof claim
 1. 15. A non-transitory carrier medium carrying data thatincludes information generated according to the computer-implementedmethod of claim
 1. 16. A computer system for processing, in an imageprocessing system, a captured scene captured of a live action scenewhile a display wall is positioned to be part of the live action scene,wherein the display wall comprises one or more structures capable ofdisplaying imagery, the computer system comprising: at least oneprocessor; and a computer-readable medium storing instructions, whichwhen executed by the at least one processor, causes the computer systemto perform operations comprising: receiving image data of the liveaction scene having a live actor and the display wall displaying a firstrendering of a precursor image, wherein the image data is captured by atleast one camera in at least one placement relative to the live actorand the display wall; determining precursor metadata for the precursorimage displayed on the display wall and display wall metadata for thedisplay wall, wherein the precursor metadata comprises pixel displaydata for display wall pixels of the display wall, and wherein thedisplay wall metadata comprises geometry data for a display wallposition relative to the at least one camera; accessing an image mattefor the image data, wherein the image matte indicates a first portionassociated with the live actor and a second portion associated with theprecursor image on the display wall in the live action scene;determining pixel display values to add or modify at least one of animage effect from the display wall pixels of the precursor image or avisual effect provided in the live action scene, and wherein the pixeldisplay values comprise one or more adjustments to pixels in the imagedata; and adjusting the image data of the captured scene using the pixeldisplay values and the image matte to add or modify the at least one ofthe image effect or the visual effect independent of rendering of theprecursor image on the display wall during capture of the image data.17. The computer system of claim 16, wherein adjusting the image data toadd or modify the at least one of the image effect or the visual effectcomprises modifying background pixels of the display wall in the imagedata based on the pixel display values.
 18. The computer system of claim17, wherein the pixel display values are determined at a time aftercapturing the image data to add the at least one of the image effect orthe visual effect to the image data based on the one or more adjustmentsindependent of re-rendering the precursor image on the display wall inthe live action scene or the image data, and wherein adjusting the imagedata to add or modify the at least one of the image effect or the visualeffect is performed at the time after capturing the image data.
 19. Thecomputer system of claim 17, wherein the pixel display values aredetermined in real-time during a capture of the image data to add the atleast one of the image effect or the visual effect to the image databased on the one or more adjustments independent of re-rendering theprecursor image on the display wall in the live action scene or theimage data, and wherein adjusting the image data to add or modify the atleast one of the image effect or the visual effect is performed inreal-time during the capture of the image data.
 20. The computer systemof claim 16, wherein, before determining the pixel display values, theoperations further comprise: determining one or more changes to theprecursor metadata for the at least one of the image effect or thevisual effect; and updating the precursor metadata based on the one ormore changes, wherein determining the pixel display values uses theupdated precursor metadata.